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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MO NTH (S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

I) S Responsive to communication(s) filed on 06 December 2004 . 
2a)D This action is FINAL. 2b)E3 This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) S Claim(s) 20-39 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) Q Claim(s) is/are allowed. 

6) S Claim(s) 20-39 is/are rejected. 

7) D C!aim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

II) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 
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a)D All b)D Some * c)D None of: 

1 Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 

1 . This action is responsive to communications: RCE and Preliminary amendments, 
both filed December 6, 2004 to the original application filed 08/09/01 . 

2. Claims 1-19 are cancelled. Claims 20 - 39 have been added. 



Continued Examination Under 37 CFR 1.114 

3. A request for continued examination under 37 CFR 1.114, including the fee set forth 
in 37 CFR 1 .17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1 .17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 
12/06/2004 has been entered. 

Claim Rejections - 35 USC § 103 

4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

This application currently names joint inventors. In considering patentability of 
the claims under 35 U.S.C. 103(a), the examiner presumes that the subject matter of 
the various claims was commonly owned at the time any inventions covered therein 



were made absent any evidence to the contrary. Applicant is advised of the obligation 
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under 37 CFR 1 .56 to point out the inventor and invention dates of each claim that was 
not commonly owned at the time a later invention was made in order for the examiner to 
consider the applicability of 35 U.S.C. 103(c) and potential 35 U.S.C. 102(e), (f) or (g) 
prior art under 35 U.S.C. 103(a). 

5. Claims 20, 27 and 34 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Non-Patent Document: "Supporting Ranked Boolean Similarity Queries in 
MARS", IEEE Trans, on Knowledge and date Engineering, 10, Nov - Dec. 1998, 
Author: Ortega et al (hereinafter "Ortega") in view of Non-Patent Document: 
"MediaNet: A multimedia Information Network for Knowledge Representation", In 
Conference on Internet Multimedia Management Systems. Vol. 4210 (numbered 
Pages 1-12), Boston, MA, Nov. 2000, 1 ST/SPIE.00, Author: Benitez et al 
(hereinafter "Benitez"). 

Regarding claims 20, 27 and 34, Ortega teaches a method for processing 
multimedia data in a computer system, comprising: 

receiving as input a high-level concept describing data to be accessed (see section 
1 .2, page 4, paragraph 1 , "A Boolean retrieval model (adapted for retrieval over images) 

is used to interpret the query being able to support such conceptual queries is 

critical for the versatility of large image databases.", and section 2, "In this section, we 
briefly describe the image features used . . . Other image features are available , 
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however we restrict ourselves to queries involving only to the above features in this 
paper."); 

translating the high-level concept into a low-level query by using stored concept 
constructs which are defined using features derived from a plurality of application 
domains (see section 1.2, page 4, paragraph 1, "A Boolean retrieval model (adapted for 

retrieval over images) is used to interpret the query being able to support such 

conceptual queries is critical for the versatility of large image databases.", and section 2, 
"In this section, we briefly describe the image features used . . . Other image features 
are available , however we restrict ourselves to queries involving only to the above 
features in this paper."). 

Ortega does not explicitly teach transferring the low-level query to one or more 
search engines and concept repository. 

Benitez teaches transferring the low-level query to one or more search engines to 
access information using the low-level query (see section 3, page 6, "Typical content- 
based retrieval systems indexed ... or the value for any low-level features in the image 

database" and section 4.2, page 9, "At this point and the results integrated into a 

unique list as described for visual queries"); and 

a concept repository for storing and accessing the concept constructs (see Fig.4 and 
section 4.1, page 7, Tor each application, the list of concepts and relationships in the 
MediaNet knowledge based should be representative of the content in the database and 
the goal of the application task."). 
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It would have been obvious to one of ordinary skill in the data processing art at the 
time of the present invention to combine teaching of the cited references because 
Benitez's teaching of "transferring the low-level query to one or more search engines to 
access information using the low-level query" and "concept repository" would have 
allowed Ortega's system to integrate both conceptual and perceptual representations of 
knowledge to impact a broad range of applications that deal with multimedia content at 
semantic and perceptual levels. This improves the performance of multimedia retrieval 
applications by using query expansion, refinement and translation across multiple 
content modalities as suggested by Benitez (see Abstract). 

6. Claims 21 - 26, 28 - 33, and 35 - 39 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Ortega in view of Benitez and further in view of Non-Patent 
Document: "CAMEL: Concept Annotated image Libraries", In Storage and Retrieval 
for Image and Video Database, San Jose, CA, Jan. 2001. SPIE (numbered pages 1 
- 12), Author: Natsev et al (hereinafter "Natsev"). 

Regarding claims 21 , 28 and 35, Ortega teaches storing matching algorithms in a 
matching algorithm library module (see section 4.8, page 20 - section 4.8.3, page 25) 
and Benitez teaches storing the concept constructs in a concept library module (see 
Fig. 4 and section 4.1 , page 7, "For each application, the list of concepts and 
relationships in the MediaNet knowledge based should be representative of the content 
in the database and the goal of the application task."). 
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Ortega or Benitez does not explicitly teach storing the features in a feature library 
module; and storing constraints in a constraint library module. 

However, Natsev further teaches storing the concept constructs in a concept library 
module (see section 1.2, "The concept cataloguing, or learning, phase is used to define 
visual concepts and build a concept library. Concepts are defined .... The concept 
library is module for a persistent storage of concepts"); 

storing the features in a feature library module (see fig.3); and 

storing constraints in a constraint library module (see section 6, "Another 
improvement that we are considering is the introduction of spatial constraints in the 
query engine. For example, . . . , the corresponding document is ranked higher"). 

It would have been obvious to one of ordinary skill in the data processing art at the 
time of the present invention to combine teaching of the cited references because 
Natsev's teaching of storing the concept constructs in a concept library module, storing 
the features in a feature library module and storing constraints in a constraint library 
module would have presented a new direction for improving current image query 
system, namely focusing the query specification part as gateway to better performance 
and usability (see Natsev section 6) for the combination Ortega and Benitez's system. 

Regarding claims 22, 29 and 36, Natsev teaches interfacing the library modules 
to the application domains (see fig.3 and section 4, "Fig.3 illustrates the architecture of 
the ... , as well as the Concept Library module"). 
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Regarding claims 23, 30 and 37, Natsev teaches building a concept construct 
(see fig.1 and section 1 .2, The concept cataloguing, or learning, phase is used to define 
visual concepts and build concept library. Concepts are defined . . . and to associate it 
with the given concept"). 

Regarding claims 24 and 31, Natsev teaches a method as defined in Claim 23, 
wherein the step of building a concept construct comprise combining one or more of the 
features with see section 1 .2, The concept cataloguing, or learning, phase is used to 
define visual concepts and build concept library) 

zero or more of the stored concept (see Fig. 1 and section 1 .2, 'The concept 
library is a module for persistent storage of concept"), and 

zero or more of the constraints (see section 6, "Another improvement that we are 
considering is the introduction of spatial constraint in query engine"). 

Regarding claims 25 and 32, Ortega teaches a concept construct is represented 
using a hierarchical fuzzy graph data tree-structure comprising: nodes that correspond 
to child-concepts and a subset of the features; aggregation edges that correspond to 
parent-child relationships; and association edges that correspond to inter-sibling 
constraints (see sections 4.1 and 4.2, pages 10 and 11). 

Regarding claims 26 and 33, Natsev teaches a method as defined in Claim 20, 
wherein the features are user defined (see section 2, "One of the most commonly used 
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image features is color histogram, . . . places the burden on the user to specify weights 
for the different features"). 

Regarding claim 38, Natsev teaches a system as defined in Claim 34, wherein 
the translation engine further comprises an interpreter that translates the high level 
concept (see fig. 1). 

Regarding claim 39, Natsev teaches a system as defined in Claim 34, further 
comprises a search engine (see section 2, "the high-level approach that . . . . , there are 
some considerations that make certain query engines more suitable than others". 

Conclusion 

7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Fred I. Ehichioya whose telephone number is 571-272- 
4034. The examiner can normally be reached on M - F 8:00 AM to 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, John E. Breene can be reached on 571-272-4107. The fax phone number 
for the organization where this application or proceeding is assigned is 703-872-9306. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

Fred I. Ehichioya 



January 7, 2005 



Patent Examiner 
Art Unit 2162 




